Cores, Debugging, and Coverage

Debugging is difficult and costly, especially for production failures. To aid developers, we enhance core memory dumps produced by crashing applications with lightweight, tunable tracing. We propose two complementary forms of tracing, path traces and global coverage, and demonstrate that they can provide substantial postmortem analysis benefit at low cost. Recent work focuses on the interplay of these mechanisms, and comparison of various forms of coverage data. 1. Problem and Motivation Testing and debugging of software is difficult, expensive, and time-consuming. Debugging, testing, and verification can account for 50–75% of a software project’s cost [12, 15, 33]. Even with extensive in-house testing, however, postdeployment failures are inevitable in complex software. In this scenario, failure reports containing traces or failurefocused views of program state are very valuable. While comprehensive reports with complete traces of productionrun failures are ideal, this level of detail is impractical for complex programs. Even for simple code, full-tracing overhead may only be acceptable during in-house testing. A core memory dump is a useful and readily-available artifact from a program crash. Coupled with symbol information, a core dump reveals the program stack for each thread at the time of termination, global variables, and some portion of heap data. Importantly, a core dump provides the crashing location and the active call-site in every other stack frame, thus providing both a snapshot of crash state and a partial call history for the program. Horwitz et al. [17] show a substantial postmortem analysis benefit by taking advantage of this information. Nevertheless, our prior work [28] indicates that significant execution ambiguity remains after postmortem analysis utilizing a stack trace alone. The goal of this work is to enhance readily-available postdeployment failure information with lightweight, tunable instrumentation. Enhanced failure information could then be given to a developer for manual debugging, or could be used to perform automated postmortem analysis to provide a reduced view of the failure state for the developer (as in this ∗ This paper originally appeared as a grand finals submission to the ACM Student Research Competition in 2014. work). Building on the information already available in a core dump can yield inexpensive but valuable postmortem data. Our existing instrumentation and analysis framework [28] validates this using two customizable core-dump-enhancement mechanisms: path traces and call-site coverage. Recent work investigates the cost and benefit of these mechanisms independently [27], and considers other forms of lightweight core dump enhancement. This document focuses on work investigating the cost/benefit trade-offs of different forms of coverage. Necessary background material is discussed in section 2. Section 3 describes our approach to lightweight instrumentation and analysis of core dump data. Section 4 discusses previous evaluations and new data comparing different granularities of coverage data. Section 5 considers related work. Section 6 concludes and suggests future and ongoing work.